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(57) Abstract: A method and apparatus are disclosed for tracking an object of interest in a digital image having at least one high 
I/") resolution portion. In a "shaped mirror embodiment," a shaped rotating mirror magnifies at least one portion of the field of view so 
£^ that a stationary camera obtains a higher resolution image corresponding to the magnified region. In a "two camera embodiment," 
J^J two stationary cameras with varying resolution are focused on a flat rotating mirror. A first stationary camera having lower resolution 
^ produces the overall panoramic image, while the second stationary camera having higher resolution produces the higher resolution 
image with a smaller field of view. In the shaped mirror embodiment, a rotating mirror contains one magnifying region that magnifies 
a portion of the field of view, and additional regions that reduce the remaining portions of the field of view. In the illustrative embodi- 
ment, the rotating mirror is curved to produce regions of variable magnifying resolution. In the illustrative two-camera embodiment, 
a 360-degree panoramic image is obtained using a first stationary camera and a flat rotating mirror. A second camera having a higher 
resolution or higher zoom than the first camera is also utilized. A semi-reflective mirror transmits a predefined percentage of the 
^ reflected light to the first camera and reflects a predefined percentage of the reflected light to the second camera. 
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Method and apparatus for tracking an object of interest in a digital image 



Field of the Invention 

The present invention relates to methods and apparatus for producing 
panoramic images, and more particularly, to a method and apparatus for tracking an object of 
interest in such panoramic images and zooming into desired portions of such panoramic 
5 images. 

Background of the Invention 

A number of techniques have been proposed or suggested for obtaining 
panoramic images from digital cameras. Generally, panoramic images have a field of view 

10 between 180 and 360 degrees. Most techniques for generating panoramic images utilize a 
number of overlapping images that are aligned using well-known image processing 
techniques to create a single, integrated image. For a general discussion of such image 
processing techniques, see, for example, P. Anandan and P.J. Burt, "Image Stabilization by 
Registration to a Reference Mosaic," ARPA94(I:425-434), BibRef 9400, or P. Anandan et 

15 al., "Real-time Scene Stabilization and Mosaic Construction," ARPA94(I:457-465), BibRef 
9400 And: A4, Al, A3, A5 and A4: WACV94(54-62), each incorporated by reference herein. 

A number of techniques have been proposed or suggested for capturing the 
plurality of overlapping images that are used to generate the panoramic image. For example, 

20 a number of systems employ multiple cameras to record the multiple images. The multiple 
images are then processed to create the desired panoramic image. Since the relationship 
between the various cameras is typically fixed, the relationship between the images is also 
known. The problem with a multiple camera arrangement, of course, is the additional 
expense relative to a single camera implementation. 

25 Thus, a number of systems obtain the plurality of images using a single 

camera. For example, International Patent Application Number WO 99/62252 discloses a 
video camera that rotates at a constant speed to capture video images of a panoramic scene of 
interest. The digital video file can then be processed to create the desired panoramic image. 



BNSDOCID: <WO. 



.0217235A2J_> 



. WO 02/17235 • PCT/EP01/09448 

2 

In another variation, Japanese Patent Application Number 1 1004373 obtains a 
360-degree panoramic image using a fixed camera and a rotating mirror. The mirror is 
mounted above the camera with a predefined tilt, such that the field of view of the mirror 
captures the desired panoramic scene as the mirror is rotated about an optical axis of the 
5 camera. The individual overlapping images are then processed to create the desired 
panoramic image. 

A number of techniques have been proposed or suggested for tracking objects 
in digital images, including panoramic images. In such tracking systems, direct tracking is 
typically performed using a low-resolution image due to computational issues, and the 
1 0 portion of the image of interest can be magnified using a higher resolution image. The 
tracking task maintains the target in the field of view of the camera so the operator can 
monitor the object of interest. In this manner, the operator can switch between a high 
resolution with a smaller field of view and a low resolution with a larger field of view, as 
desired. 

1 5 The higher resolution image is typically obtained using a moving part, such as 

an additional mirror or camera, whose motion is controlled during a zoom operation, to 
magnify the region of interest. For example, Mitsubishi Electric America provides an omni- 
directional vision system utilizing two mirrors. See, for example, 
http://www.mitsubishi.com/mea/tecpmn.html. In an implementation that utilizes an 

20 additional mirror to magnify the area of interest, complex optical arrangements are typically 
required. Likewise, in an implementation that utilizes an additional camera to magnify the 
area of interest, the increased expense over a single camera implementation is prohibitive. 

A need therefore exists for a method and apparatus that provides tracking in a 
panoramic image without controllable moving hardware for magnification. A further need 

25 exists for a method and apparatus that provides tracking in a panoramic image with 
simultaneous magnification of multiple portions of the field of view, without the 
mechanically controlled motion of additional moving parts. 

Summary of the Invention 
30 Generally a method and apparatus are disclosed for tracking an object of 

interest in a digital image having at least one high resolution portion. In one embodiment, 
referred to herein as the "shaped mirror embodiment," a shaped rotating mirror magnifies at 
least one portion of the field of view so that a stationary camera obtains a higher resolution 
image corresponding to the magnified region. In a second embodiment, referred to herein as 
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the "two camera embodiment/' two stationary cameras with varying resolution are focused 
on a flat rotating mirror. A first stationary camera having lower resolution produces the 
overall panoramic image, while the second stationary camera having higher resolution 
produces the higher resolution image with a smaller field of view. 

5 In the shaped mirror embodiment, a rotating mirror contains one magnifying 

region that magnifies a portion of the field of view, and one or more additional region(s) that 
reduce the remaining portions of the field of view. In the illustrative embodiment, the rotating 
mirror is curved to produce regions of variable magnifying resolution. The magnifying region 
that magnifies a portion of the field of view has a generally concave shape while reduction 

10 regions that reduce the remaining portions of the field of view have a convex shape. 

In the illustrative two-camera embodiment, a 360-degree panoramic image can 
be obtained using a first stationary camera and a flat rotating mirror. The flat rotating mirror 
rotates about an optical axis of the camera. The mirror is mounted above the camera with a 
predefined tilt, such that the field of view of the mirror captures the desired panoramic scene 

15 as the mirror rotates. A second camera having a higher resolution or higher zoom than the 
first camera is also utilized. A second, semi-reflective mirror is positioned along the optical 
axis. The semi-reflective mirror transmits a predefined percentage of the reflected light to the 
first camera and reflects a predefined percentage of the reflected light to the second camera. 

A more complete understanding of the present invention, as well as further 

20 features and advantages of the present invention, will be obtained by reference to the 
following detailed description and drawings. 

Brief Description of the Drawings 

FIG. 1 illustrates a panoramic image tracking system in accordance with one 
25 embodiment of the present invention; 

FIGS. 2 A and 2B illustrate a front elevation view and a top elevation view, 
respectively, of one embodiment of the rotating mirror of FIG. 1; 

FIG. 3 illustrates a panoramic image tracking system in accordance with a two 
camera embodiment of the present invention; and 
30 FIG. 4 is a flow chart describing an exemplary panorama tracking process 

incorporating features of the present invention. 



BNSDOCID: <WO 021 7235A2_I_> 



. WO 02/17235 • PCT/EP01/09448 

4 

Detailed Description 

FIG. 1 illustrates a panoramic image tracking system 100 in accordance with 
one embodiment of the present invention. The present invention obtains a 360-degree 
panoramic image using a stationary camera 120 and a rotating mirror 200 5 discussed further 
5 below in conjunction with FIG. 2. In the illustrative embodiment described herein, the 

> 

stationary camera 1 20 is suspended in a room, defined by a ceiling 110 and two walls 115a, 
1 15b. As shown in FIG. 1, the rotating mirror 200 rotates about an optical axis 130 of the 
camera 120. The mirror 200 is rotated using a passive, non-controllable motor that rotates 
about 360 degrees at a fixed speed. The mirror 200 is mounted above the camera 120 with a 

10 predefined tilt, such that the field of view 140 of the mirror 200 captures the desired 
panoramic scene as the mirror 200 is rotated about the camera 120. In this manner, the 
panoramic image tracking system 100 processes a sequence of images that can be processed 
using well-known image processing techniques to create the desired panoramic image. 

The present invention provides a panoramic image having at least one high- 

15 resolution portion. According to one feature of the present invention, tracking is performed in 
a panoramic image without controllable moving hardware. In one embodiment, referred to 
herein as the "shaped mirror embodiment," a shaped rotating mirror 200 magnifies at least 
one portion of the field of view so that the stationary camera 120 obtains a higher resolution 
image corresponding to the magnified region. In another embodiment, referred to herein as 

20 the "two camera embodiment/' two stationary cameras 1 20 with varying resolution are 

focused on a flat rotating mirror 200. A first stationary camera 120-1 having lower resolution 
produces the overall panoramic image, while the second stationary camera 120-2 having 
higher resolution produces the higher resolution image with a smaller field of view. . 

Shaped Mirror Embodiment 

25 FIG. 2A is a front elevation view of an illustrative rotating mirror 200 in 

accordance with the present invention. As shown in FIG. 2A, the illustrative rotating mirror 
200 contains one magnifying region 220 that magnifies a portion of the field of view, and 
two regions 210a, 210b that reduce the remaining portions of the field of view. Thus, the 
illustrative rotating mirror 200 shown in FIG. 2A rotates along the horizontal axis, and the 

30 magnifying region 220 produces a vertical high-resolution band in each image. 

FIG. 2B is a top elevation view of the illustrative rotating mirror 200 of FIG. 
2A. As shown in FIG. 2B, the rotating mirror 200 is curved to produce regions of variable 
magnifying resolution: The magnifying region 220 that magnifies a portion of the field of 
view has a generally concave shape while reduction regions 210a, 210b that reduce the 
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remaining portions of the field of view, have a convex shape. In other words, the magnifying 
region 220 is a high-resolution area that is obtained by magnifying the horizontal dimension 
and projecting a portion 240 of the field of view to an image strip wider than the field of view 
portion 240. Likewise, the reduction regions 210a, 210b are low resolution areas that are 
5 obtained by shrinking the horizontal dimension and projecting the corresponding portions 
230a, 230b of the field of view to an image strip narrower than the field of view portions 
230a, 230b. Generally, the horizontal resolution in the image of portions 230a, 230b will be 
f/a pixels per panoramic degree, whereas the horizontal resolution in the image of portion 
240 will be fb pixels per panoramic degree, where f is a constant and both a and b are greater 
1 0 than one. 

Thus, as the rotating mirror 200 rotates, the stationary camera 120 captures the 
plurality of images that are used to generate the desired panoramic image. Each image 
captures the entire field of view 140 (shown in FIGS. 1 and 2B), comprised of a high- 
resolution portion 240 and two low resolution portions 230a, 230b. According to another 

1 5 feature of the present invention, the high resolution portion 240 is sub-sampled to achieve the 
same resolution as the two low resolution portions 230a, 230b. Specifically, the high 
resolution portion 240 is sub-sampled by a factor of 1/ab. 

In this manner, the desired panoramic image is obtained as a low resolution 
image. Thus, moving objects appear and may be tracked in the single low-resolution image. 

20 For a detailed discussion of the tracking of objects of interest in a low-resolution image, see, 
for example, D. Gutchess et al., "Automatic Surveillance Using Omni-Directional and Active 
Cameras" Proc. Asian Conference on Computer Vision 2000 (2000), or C. Wren et al., 
"Pfinder: Real-Time Tracking of the Human Body", IEEE. Trans. PAMI, 19(7) (1997), each 
incorporated by reference herein. 

25 The rotation speed of the mirror 200 and the horizontal width of the vertical 

band 240 are coordinated for a given implementation, such that a high resolution image is 
obtained for each portion of the overall field of view 140 for each rotation cycle of the mirror 
200. In this manner, the panoramic image is updated with each cycle of the mirror rotation. A 
low resolution version of the entire panorama is maintained, as well as each of the adjacent 

30 high resolution vertical bands 240 corresponding to full panorama. 

As discussed further below in conjunction with FIG. 4, given a request to 
magnify a portion of the field of view, the corresponding high resolution vertical strip 240 is 
retrieved, and super-imposed on the low resolution image of the entire panorama. It is noted 
that the present invention allows any resolution between the low-resolution image and the 
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high-resolution image by sub-sampling the high-resolution image to the desired resolution. 
The field of view, however, for anything above the low resolution image is the smaller field 
of view of the high resolution image. 

Two-Camera Embodiment 
5 FIG. 3 illustrates a panoramic image tracking system 300 in accordance with 

the two-camera embodiment of the present invention. The two-camera embodiment obtains a 
360-degree panoramic image using a first stationary camera 320-1 and a flat rotating mirror 
325. The first stationary camera 320-1 is suspended in a room, defined by a ceiling 310 and 
two walls 315 a, 315b. As shown in FIG. 3, the flat rotating mirror 325 rotates about an 

10 optical axis 330 of the camera 320-1 . The mirror 325 is rotated using a passive, non- 
controllable motor that rotates about 360 degrees at a fixed speed. The mirror 325 is mounted 
above the camera 320-1 with a predefined tilt, such that the field of view 340 of the mirror 
325 captures the desired panoramic scene as the mirror 325 is rotated about the camera 320- 
1 . In this manner, the panoramic image tracking system 300 processes a sequence of images 

1 5 that can be processed using well-known image processing techniques to create the desired 
panoramic image. 

In the two-camera embodiment, a second camera 320-2 having a higher 
resolution or higher zoom than the first camera 320-1 is also utilized. As shown in FIG. 3, a 
second, semi-reflective mirror 345, such as a half-silvered mirror, is positioned along the 

20 optical axis 330. The semi-reflective mirror 345 transmits a predefined percentage of the 
reflected light from mirror 325 to camera 320-1 and reflects a predefined percentage of the 
reflected light from mirror 325 to camera 320-2. The first camera 320-1 captures a series of 
overlapping low-resolution images that can be integrated to obtain the panoramic image. In 
addition, the second camera 320-2 captures a series of higher resolution images that can be 

25 utilized to zoom into a desired area of interest.. In the illustrative embodiment, the higher 
resolution images captured by the second camera 320-2 are vertical bands, in a similar 
manner to the shaped mirror embodiment discussed above. 

Tracking Process 

The present invention can track an object of interest in the single low- 
30 resolution image of the panorama. FIG. 4 is a flow chart describing an exemplary panorama 
tracking process 400 incorporating features of the present invention. As shown in FIG. 4, the 
panorama tracking process 400 continuously captures a panoramic image with low resolution 
and strips of high resolution using the shaped mirror or two camera embodiments discussed 
above during step 410. 
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Once a request is received to magnify a portion of the field of view (for 
example, from an automated tracking module following a moving object), as detected during 
step 41 5, the panorama tracking process 400 retrieves the corresponding high-resolution 
vertical strip 240 during step 420. The retrieved high-resolution vertical strip 240 is then 
super-imposed during step 430 on the low-resolution image of the entire panorama. It is 
again noted that the present invention allows any resolution between the low-resolution 
image and the high-resolution image by sub-sampling the high-resolution image to the 
desired resolution. Program control then returns to step 410 to continue updating the 
panoramic image in real-time, until a new magnification region is selected. 

It is to be understood that the embodiments and variations shown and 
described herein are merely illustrative of the principles of this invention and that various 
modifications may be implemented by those skilled in the art without departing from the 
scope and spirit of the invention. 
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1 . A digital image system (100), comprising: 

a shaped rotating mirror (200) having a field of view (140), wherein said field 
of view (140) includes a region of higher resolution (220); and 

a camera (120) focused on said rotating mirror (200) and capturing a plurality 
5 of images of said field of view (140). 



10 



2. The digital image system (100) of claim 1, wherein said shaped rotating mirror 

(200) includes a generally concave region for obtaining said region of higher resolution (220) 
of said field of view (140). 



3. The digital image system (100) of claim 1, wherein said shaped rotating mirror 

(200) includes at least one generally convex region (210) for obtaining a region of lower 
resolution of said field of view (140). 



1 5 4. The digital image system (100) of claim 1, wherein said shaped rotating mirror 

(200) rotates at a fixed speed. 



5. The digital image system (100) of claim 1, wherein said shaped rotating mirror 

(200) is rotated using a passive, non-controllable motor that rotates said mirror about a fixed 
20 axis. 



6. The digital image system (100) of claim 1, wherein said field of view (140) 

captures a panoramic image. 



25 7. A digital image system (100), comprising: 

a rotating mirror (325) having a field of view (340); 

a first stationary camera (320-1) focused on said rotating mirror (325) and 
capturing a plurality of images of said field of view (340); 



BNSDOCID: <WO 



0217235A2 I > 



WO 02/17235 * PCT/EPO 1/09448 

9 

a second mirror (345) in an optical path before said first stationary camera 

(320-1); and 

a second stationary camera (320-2) focused on said second mirror (345) and 
capturing a plurality of images of said field of view (340), wherein said second camera (320- 
5 2) has a higher resolution than said first camera (320-1).. 

8. The digital image system (100) of claim 7, further comprising a semi- 
reflective mirror positioned in an optical path of said first and second stationary cameras 
(320-1,2) such that said semi-reflective mirror transmits a predefined percentage of light 

1 0 reflected light from said rotating mirror (325) to said first camera (320-1) and reflects a 
predefined percentage of said light reflected from said rotating mirror (325) to said second 
camera (320-2). 

9. The digital image system (100) of claim 7, wherein said field of view (340) 
1 5 captures a panoramic image. 

10. The digital image system (100) of claim 7, wherein said rotating mirror (325) 
rotates at a fixed speed. 

20 11. The digital image system (100) of claim 7, wherein said field of view (140) 

captures a panoramic image. 

12. The digital image system (100) of claim 7, wherein said second mirror (345) is 
a semi-reflective mirror. 

25 

13. The digital image system (100) of claim 7, wherein said second mirror (345) is 
a half-silvered mirror. 

14. A method for obtaining a digital image, comprising the steps of: 

30 rotating a shaped mirror (200) having a field of view (140), wherein said field 

of view (140) includes a region of higher resolution (220); and 

focusing a camera (120) on said rotating mirror (200); and 
capturing a plurality of images of said field of view (140). 
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15. The method of claim 14, wherein said shaped mirror (200) includes a 

generally concave region for obtaining said region of higher resolution (220) of said field of 
view (140). 

5 16. The method of claim 14, wherein said shaped mirror (200) includes at least 

one generally convex region (210) for obtaining a region of lower resolution of said field of 
view (140). 

17. The method of claim 14, wherein said rotating step rotates said shaped mirror 
10 (200) at a fixed speed. 

1 8 . The method of claim 1 4, wherein said rotating step rotates said shaped mirror 
(200) using a passive, non-controllable motor that rotates said mirror about a fixed axis. 

15 19. The method of claim 14, wherein said field of view (140) captures a 

panoramic image. 

20. The method of claim 19, further comprising the step of integrating said 
plurality of images to obtain said panoramic image. 

20 

21. A method for obtaining a digital image, comprising the steps of: 
rotating a mirror (325) having a field of view (340); 

focusing a first stationary camera (320-1) on said rotating mirror (325); 
capturing a plurality of images of said field of view (340) using said first 
25 stationary camera (320-1); 

positioning a second mirror (345) in an optical path before said first stationary 
camera (320-1); and 

focusing a second stationary camera (320-2) on said second mirror (345) and 
capturing a plurality of images of said field of view (340), wherein said second camera (320- 
30 2) has a higher resolution than said first camera (320-1). 

22. The method of claim 21, further comprising the step of positioning a semi- 
reflective mirror (345) in an optical path of said first and second stationary cameras (320-1,2) 
such that said semi-reflective mirror transmits a predefined percentage of light reflected light 
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from said rotating mirror (325) to said first camera (320-1) and reflects a predefined 
percentage of said light reflected from said rotating mirror (325) to said second camera 
(320-2). 

5 23. The method of claim 2 1 , wherein said plurality of images can be integrated to 

obtain a panoramic image. 

24. The method of claim 21 5 wherein said rotating step rotates said mirror (325) at 
a fixed speed. 

10 

25. The method of claim 21, wherein said second mirror (345) is a semi-reflective 
mirror. 

26. The method of claim 21, wherein said second mirror (345) is a half-silvered 
1 5 mirror. 

27. An article of manufacture for obtaining a digital image, comprising: 

a computer readable medium having computer readable code means embodied thereon, said 
computer readable program code means comprising: 
20 a step to rotate a shaped mirror (200) having a field of view (140), wherein said field of view 
(140) includes a region of higher resolution (220); and 
a step to focus a camera on said rotating mirror (200); and 
a step to capture a plurality of images of said field of view (140). 

25 28. An article of manufacture for obtaining a digital image, comprising: 

a computer readable medium having computer readable code means embodied thereon, said 

computer readable program code means comprising: 

a step to rotate a mirror (325) having a field of view (340); 

a step to focus a first stationary camera (320-1) on said rotating mirror (325); 
30 a step to capture a plurality of images of said field of view (340) using said first stationary 

camera (320-1); 

a step to position a second mirror (345) in an optical path before said first stationary camera 
(320-1); and 
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* 

a step to focus a second stationary camera (320-2) on said second mirror (345) and capture a 
plurality of images of said field of view (340), wherein said second camera (320-2) has a 
higher resolution than said first camera (320-1). 
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resolution or higher zoom than the first camera is also utilized. A semi-reflective mirror transmits a predefined percentage of the 
reflected light to the first camera and reflects a predefined percentage of the reflected light to the second camera. 
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